DOCUMENT RESUME 



ED 476 868 



TM 034 968 



AUTHOR 

TITLE 

SPONS AGENCY 
PUB DATE 
NOTE 

PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Um, Eunkyoung; Dogan, Enis; Im, Seongah; Tatsuoka, Kimumi; 
Corter, James E. 

Comparing Eighth-Grade Diagnostic Test Results for Korean, 
Czech, and American Students. 

National Science Foundation, Arlington, VA. 

2003-04-22 

23p. 

Reports - Research (143) 

EDRS Price MFOl/PCOl Plus Postage. 

Achievement Tests; Cross Cultural Studies; Diagnostic Tests; 
Foreign Countries; International Studies; ^Knowledge Level; 
^Mathematics Achievement ; * Student Characteristics ; *Test 
Items 

Czech Republic; Korea; *Third International Mathematics and 
Science Study; United States 



ABSTRACT 

Diagnostic analyses were conducted on data from the Third 
International Mathematics and Science Study second population (TIMSS-R; 1999) 
from the United States, Korea, and the Czech Republic in terms of test item 
attributes (i.e., content, processing skills, and item format) and inferred 
students’ knowledge. The Rule Space model (K. Tatsuoka, 1998) was used to 
infer student mastery of individual attributes. Korean, U.S., and Czech 
students ere compared in terms of patterns of attribute mastery. First, 
attribute mastery probabilities were related to their overall mathematics 
score (achievement level). Second, profiles of mean attribute mastery 
probabilities were compared between Czech, Korean, and U.S. students. 

Students of these three countries show relatively high achievement in integer 
arithmetic skills, using figures and graphs, evaluation/verification, 
proportional reasoning, translation, judgmental applications of knowledge in 
geometry, and computational applications of knowledge in geometry. These 
students are relatively low in statistics /data analysis, unit conversion, and 
applying and evaluating mathematical correctness. Finally, the effects of 
student characteristics on patterns of mathematics achievement were 
investigated in the United States, Korea, and the Czech Republic. Students' 
self-concept in mathematics and socioeconomic composition are significant 
predictors of student achievement in geometry, open-ended items, and logical 
reasoning in all three countries. An appendix lists knowledge and process 
skills related to achievement on TIMSS-R. (Contains 12. tables and 7 
references.) (SLD) 



Reproductions supplied by EDRS are the best that can be mad; 
from the original document 



i 



TM034968 ed 476 868 



April 22, 2003 



Comparing Eighth-Grade Diagnostic Test Results for 
Korean, Czech, and American Students 

Eunkyoung Um 
Enis Dogan 
Seongah Im 
Kikumi Tatsuoka 
James E. Corter 

Columbia University, Teachers College 



This study has been supported by the National Science Foundation. 

ABSTRACT 



Diagnostic analyses were conducted of TIMSS-R data from the U.S., Korea, and the Czech 
Republic in terms of test item attributes (i.e., content; processing skills; and item fonnat) and 
inferred students’ knowledge. The Rule Space model (Tatsuoka, 1998) was used to infer student 
mastery of individual attributes. Korean, U.S., and Czech students were compared in tenns of 
patterns of attribute mastety. First, attribute masteiy probabilities were related to their overall math 
score (achievement level). Second, profiles of mean attribute masteiy probabilities were compared 
between Czech, Korean, and U.S. students. Students of these three countries show relatively high 
achievement in integer arithmetic skills, using figures and graphs, evaluation/verification, 
proportional reasoning, translation, judgmental applications of knowledge in geometiy, and 
computational applications of knowledge in geometry. These students are relatively low in 
statistics/data analysis, unit conversion, and applying and evaluating mathematical correctness. 
Finally, the effects of student characteristics on patterns of mathematics achievement were 
investigated in the U.S., Korea, and the Czech Republic. Students’ self-concept in mathematics & 
socioeconomic composition are significant predictors of student achievement in geometry, open- 
ended item, and logical reasoning in the U.S., Korea, and the Czech Republic. 
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I. Overview 



The Third Iiitematioiial Math and Science Study, or TIMSS is a collaborative research project 
sponsored by the Inteniatioiial Association for the Evaluation of Educational Achievement (lEA). A 
successor to the acclaimed 1995 TIMSS, the TIMSS 1999 focused on the mathematics and science 
achievement of eighth -grade students (Cozzens, 2001). Tatsuoka identified the knowledge and 
cognitive processing skills that are required to answer the mathematics items in the TIMSS 1999 
test administered to eighth grade students in the United States (Tatsuoka, 2001, Appendix 1). The 
hypothesized skills were tested statistically via the Rule Space methodology* (Tatsuoka, 1998). 

In this study, we analyzed the achievement test data from U.S., Korean, and Czech Republican 
segments of Population 2 of TIMSS in tenns of attributes (latent variables such as hiowledge and 
cognitive processing skills), knowledge states, and the structures of knowledge states. The primary 
underlying motivation for our research prompt the following inquiries.* 1) “What similarities and 
differences emerge among students from the U.S., Korea, and the Czech Republic in ternis of 
mastery probability attributes?” 2) “What are the effects of student background characteristics on 
the eighth grade mathematics achievement in the U.S., Korea, and the Czech Republic in the areas 
of content, process, and skill attributes?” 

In order to address these questions, the outcome of the rule space analysis for each student was used 
for the first phase of research. First, attribute mastery probabilities were related to each student’s 
overall math score. Second, profiles of mean attribute mastery probabilities were compared 
between U.S, Korean, and Czech Republic students. 

The profiles (Figures 1-3) show that students of the three countries show relatively high 
achievement in integer arithmetic skills, using figures and graphs, evaluation/verification, 
proportional reasoning, translation, judgmental applications of knowledge in geometry. The profiles 
show that these students are low in statistics/data analysis, unit conversion, and applying and 
evaluating mathematical correctness. The largest differences between the U.S. and Korea are in 
geometry; data management, patterns, and open-ended item. The largest differences between the 
U.S. and the Czech Republic are in geometry, logical reasoning, and open-ended item. 



' Rule Space methodology' estimates the probability of correctly applying each skill for each individual. 
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TIMSS items for the second phase of research were selected based on a student’s ability to 
delineate proposed underlying constructs associated with his or her background. We used the 
outcome of the rule space analysis for each student to see if policy variables such as students’ self- 
concept or socioeconomic composition are significant predictors of student achievement. Tree 
regression procedure with jackknife process was applied to reduce this set with 531 student 
background variables to a manageable size. Regression analyses were used to relate student 
background to student achievement in geometry, open-ended item, and logical reasoning. Students’ 
self-concept in mathematics & socioeconomic composition are significant predictors of student 
achievement in geometry, open-ended item, and logical reasoning in the U.S., Korea, and the Czech 
Republic. 

We believe this study can be beneficial in improving the quality of classroom instruction in the U.S., 
Korea, and the Czech Republic and help in the areas of accountability and policymaking. Our hope 
is that this study can add valuable insights into the influence of educational policy on mathematics 
achievement in the U.S., Korea, and the Czech Republic. 



11. Introduction and Background 

The Third International Math and Science Study, or TIMSS is a collaborative research project 
sponsored by the International Association for the Evaluation of Educational Achievement 
(Cozzens, 2001). TIMSS assessed the mathematic and science performance of U.S. students in 
comparison to their peers in other nations at three different grade levels in 1995 and at one grade 
level in 1999 (Cozzens, 2001). A successor to the acclaimed 1995 TIMSS, the TIMSS 1999 
focused on the mathematics and science achievement of eighth -grade students (Cozzens, 2001). 
TIMSS 1999 included 38 countries. The 1999 assessment collected extensive infonnation from 
students, teachers, and school principals about mathematics and science curricula, instruction, home 
contexts, and school characteristics and policies (Cozzens, 2001). Of the 38 participating countries, 
26 of them also participated in the 1995 TIMSS assessment, which enabled these countries to look 
for trends in their children’s mathematic and science achievement over tlie course of several years 
(Cozzens, 2001). 

In 1999, U.S. eighth -graders’ average scores were higher than the international average in math 
among the 38 participating nations (Cozzens, 2001). U.S. eighth-graders exceeded the international 
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average and, performed similarly to students in six other nations. Among the top performing 
nations in 1999 were five Asian industrialized countries - Singapore, Korea, Chinese Taipei, Hong 
Kong SAR, and Japan (Cozzens, 2001). 

Nine percent of U.S eighth-graders scored 616 or higher, placing them among the top 10 percent of 
all eighth-graders in the 38 nations participating in 1999 (Cozzens, 2001). U.S. eighth-graders’ 
average scores were higher than the international average in three of the five mathematics content 
areas assessed in 1999: fractions and number sense, data representation, analysis, and probability 
and algebra (Cozzens, 2001). 

In contrast to traditional psychometric approaches, Tatsuoka (1998) has developed a new model 
and methodology (the Rule Space method) to explain the proficiency of students on large-scale 
assessments, and she has applied the method to several tests. The Rule Space method (RSM) 
enables us to relate traditional scale scores to the underlying knowledge states (Tatsuoka, 1998). 
Tatsuoka identified knowledge and cognitive processing skills that are required to answer the 
mathematics items in the TIMSS 1999 test administered to eighth grade student in the United States 
(Tatsuoka, 2001, Appendix 1). Tlie hypothesized skills that were represented in a “Q” matrix^ were 
tested statistically via the RSM (Tatsuoka, 1998). 

Tlie RSM deals with any latent variables such as knowledge and cognitive processing skills 
(Tatsuoka, 2001). These latent variables are called “attributes” (Tatsuoka, 2001, Appendix 1). RSM 
is a statistical tool for hypothesis testing of such generated attributes. If the hypotheses are right, 
RSM will confinn the importance of the attributes and mastery of the attributes for individual 
students (Tatsuoka, 1998). 

The primary underlying motivation and driver for our research prompted the following inquiries.* 1) 
“What similarities and differences emerge among students from the U.S., Korea, and the Czech 
Republic in the content, process, and skill attributes?” 2) “What are the effects of student 
background characteristics on eighth grade mathematics achievement in the U.S., Korea, and the 
Czech Republic in the content, process, and skill attributes?” 



^ Q matrix: qjk = 1 if item j involves skill k, else qjk = 0 
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III. Method 



In order to address these questions, the outcome of the rule space analysis for each student was used 
for the study. The proposed samples, one classroom of eighth grade students per school and 
associated mathematics teachers and administrators, were drawn from U.S., Korean, and Czech 
Republican segments of Population 2 of TIMSS (Mullis, 1999). 

Goal I. 

To classify the knowledge states of U.S., Korean, and Czech Republican eighth-grade students 

First, attribute mastery probabilities were related to overall math scores (achievement level). 

Second, profiles of mean attribute mastery probabilities were compared between U.S., Korean, and 
Czech Republican students. 

Goal II. 

To evaluate the effects of student background characteristics on the eighth grade mathematics 
achievement in the U.S, Korea, and the Czech Republic in the attributes 

We used the outcome of the rule space analysis for each student to see if policy variables such as 
students’ self-concept or socioeconomic composition are significant predictors of student 
achievement. Tree regression procedure with jackknife process was applied to reduce this set with 
531 student background variables to a manageable size. This process provides a regression model 
that would predict attribute mastery probabilities estimated in the rule space analysis by using 
student background variables as predictors. After tree analyses with Jackknife process, the number 
of times that a predictor is selected as a significant variable in whole process is counted as an index 
to measure its importance on predicting the achievement of each attribute. In the study, if the 
number of selections as a significant predictor is more 90 percent of the time, the predictor is 
selected as a strong predictor. 

The resulting set of variables for the U.S., Korea, and the Czech Republic was more manageable 
compared to the set with 531 variables. Below is a table that summarizes how many variables were 
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found to be significant predictors of achievement scores across the U.S., Korea, and the Czech 
Republic for three specific attributes that we focus more on in the following sections. 

Table 1. Number of significant predictors of achievement 





Geometry 


Open-ended Item 


Logical Reasoning 


US 


20 


65 


55 


Korea 


33 


43 


47 


Czech 


11 


41 


38 



These results served as a starting point for the further analyses to make comparisons among U.S., 
Korean, and Czech students. 

When attribute mastery profiles of the U.S., Korea, and the Czech Republic are compared, there are 
big differences in average probabilities of the attributes. As seen from figures the largest differences 
across three countries are selected, attribute C4 (basic concepts and properties of two-dimensional 
geometry) of content domain, SIO (logical reasoning) of skill domain, and P5 (open-ended 
questions) of process domain. 

A three-step procedure was performed. Firstly, the set of predictors from tree regression for each 
country was combined in order to detemiine the set of student background variables that are strong 
predictors of attribute mastery probabilities for any of these three countries. As a second step, after 
detennining this set of variables, linear regressions on three attributes C4, SIO, and P5 are 
employed for each country separately with the combined predictors. Selected are the common 
significant predictors of the U.S. and Korea, and of the U.S. and Czech Republic. As a last step. 
Hostelling procedures were used to identify if there were significant differences in common 
predictors between U.S and Korean and between U.S. and Czech Republic. 

IV. Validation of the Attributes 

Rule space analysis was perfonned on nine different countries including the U.S., Korea, and the 
Czech Republic by using the content, process, and skill attributes. 
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Results of the rule space analysis are as follows: using a < 4.5 criterion for classifying a 
respondent’s item response pattern into the nearest knowledge state, an average 99.9% 
classification rate across nine countries. In other words, almost all observed item response patterns 
could be classified into one of the logically derived knowledge states. This demonstrated that the 
set of attributes listed above could explain students’ underlying perfomiances on TIMSS items for 
eighth graders very well. 

In an additional attempt to validate the attributes we also performed regression analysis using 
attributes as predictors of students’ total scores. The results were quite strong. Table 2 below shows 
the resulting R-square values for the U.S., Korea and the Czech Republic. 



Table 2. The R-square values for the U.S., Korea and the Czech Republic in the regression 
equations of predicting total scores by attribute mastery probabilities. 





USA 


Korea 


Czech 


R^-values 


.97 


.98 


.98 


SD.E 


.04 


.02 


.03 



As a result both the RSM classification rates and the high R-square values obtained from the 
regression analysis described above served as a validation for the attributes we used for the 
analysis. 
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V. Rule Space Knowledge Profiles 



Profiles of mean attribute mastery probabilities were compared for Korean, U.S., and Czech 
Republic students (Table 3). This analysis was perfonned on the data pooled across all booklets for 
comparing the total scores. For each country, estimated attribute mastery profiles were computed. 



ATTRIBUTES 


MASTERY 


MASTERY 


MASTERY 




PROBABILITY 


PROBABILITY 


PROBABILITY 




(Czech 

Republic) 


(U.S.) 


(KOREA) 


C 1 : Basic concepts in Integers 


0.93* 


0.95* 


0.93* 


C2: Basic concepts in fraction 


0.86 


0.98 


0.97 


C3: Basic concepts in algebra 


0.76 


0.84 


0.75 


C4: Basic concepts of Geometry 


0.68 


0.98 


0.94 


C5: Data and statistics 


0.73 


0.75 


0.75 


C6: Measurement 


0.63 


0.69 


0.75 


S 1 : Unit conversion 


0.38 


0.45 


0.52 


S2: Number sense/number line 


0.78 


0.82 


0.81 


S3: Using figures & graphs 


0.95* 


1.00* 


0.99* 


S4: Approximation/Estimation 


0.88 


0.82 


0.88 


S5: Evaluate/Verify/Check Options 


0.97* 


1.00* 


1.00* 


S6: Patterns & relationships 


0.46 


0.71 


0.66 


S7: Using proportional reasoning 


0.91* 


0.97* 


0.90* 


S8: Solving NovelAJnfamiliar problems 


0.87 


0.91 


0.95 


SIO: Open-ended item 


0.60 


0.85 


0.82 


SI 1: Word Problem 


0.88 


0.96 


0.96 


PI: Translate/fonnulate equations 


0.95* 


0.97* 


0.98* 


P2: Computational applications of knowledge 
in geometry 


0.92* 


0.98* 


0.96* 


P3: Judgmental applications of know ledge in 
arithmetic and geometry 


0.85* 


0.97* 


0.91* 


P4: Applying rules in algebra 


0.59 


0.76 


0.65 


P5: Logical reasoning 


0.65 


0.84 


0.82 


P6: Problem Search 


0.83 


0.93 


0.92 


P7: Generating Figures 


0.77 


0.93 


0.94 


P8: Applying & Evaluating mathematical 
correctness 


0.57 


0.59 


0.55 


P9: Management of Data & Procedures 


0.68 


0.95 


0.81 


PIO: Quantitative & Logical Reading 


0.87 


0.88 


0.82 



*High achievement 



Table 3. Mean Attribute Mastery Probabilities 
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These group profiles are plotted in Figures 1 (Content attributes), 2 (Skill), and 3 (Process). 

The content profiles (Figure 1) show that all three students show relatively high achievement in 
integer arithmetic skills. The profiles show that these students are low in statistics/data analysis. 
The profiles show that the largest difference between the U.S. and Korea is in the content area of 
geometry. Profiles show that the largest difference between the U.S. and the Czech Republic 
students is in the content area of geometry. This probably reflects the fact that achievement in these 
topics is more a function of opportunity to learn, rather than of student ability (Tatsuoka, 1998). In 
interpreting these results, it is important to consider the math content areas and topics that students 
have likely encountered in the years leading up to and including eighth grade (Cozzens, 2001). 



Figure 1. Content Attribute Mastery Probabilities 
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Table 4. Content Attribute Mastery Probabilities 



Attribute 


U.S. 


Korea 


Czech 


Korea- 

U.S. 


Czech- 

U.S. 


C 1 : Basic concepts in Integers 


0.93* 


0,95* 


0.93* 


0.02 


0,00 


C2: Basic concepts in fraction 


0.86 


0,98 


0,97 


0.12 


o.n 


C3: Basic concepts in algebra 


0.76 


0.84 


0,75 


0.08 


-0,01 


C4: Basic concepts of Geometry 


0.68 


0.98 


0.94 


0.30 


0.26 


C5: Data and statistics 


0.73 


0,75 


0.75 


0,02 


0.02 


C6: Measurement 


0,63 


0,69 


0.75 


0,06 


0.12 



*High achievement 
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Tlie skill profiles (Table 5; Figure 2) show that students of all three countries show high 
achievement in using figures, tables, charts, and graphs, evaluation/verification, and using 
proportional reasoning. The profiles show that these students are low in unit conversion. The 
profiles show that the largest differences between the U.S. and Korean students are in recognizing 
patterns and open-ended item. Profiles show that the largest difference between the U.S. and the 
Czech Republic students is in open-ended item. Tliese item types may tap more abstract thinking 
skills, rather than rote perfomiance (Tatsuoka, 1998). 

Figure 2. Skill Attribute Mastery Probabilities 
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Table 5. Skill Attribute Mastery Probabilities 



Attribute 


Description 


U.S. 


Korea 


Czech 

Republic 


Korea- 

U.S. 


Czech 

Republic 

-U.S. 


SI p 


SI: Unit conversion 


0,38 


0.45 


0,52 


0.07 


0.14 


S2 p 


S2: Number sense/number line 


0,78 


0.82 


0,81 


0.04 


0.03 


S3 p 


S3: Using figures & graphs 


0.95^ 


l.OO^ 


0.99^ 


0.05 


0.04 


S4 p 


S4: Approximation/Estimation 


0.88 


0.82 


0,88 


-0.06 


0.00 


S5 p 


S5: EvaluateA^erify/Check Options 


0,97^ 


1.00* 


1.00* 


0.03 


0.03 


S6 p 


S6: Patterns & relationships 


0.46 


0.71 


0,66 


0.25 


0.20 


S7 p 


S7: Using proportional reasoning 


0.91* 


0.97* 


0.90* 


0.06 


0.01 
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S8 p 


S8: Solving Novel/Unfamiliar 
problems 


0.87 


0.91 


0.95 


0.04 


0.08 


S10_p 


SIO: Open-ended item 


0.60 


0.85 


0.82 


0.25 


0.22 


S11.J3 


Sll: Word Problem 


0.88 


0.96 


0.96 


0.08 


0.08 



*High achievement 



The process profiles (Table 6; Figure 3) show that all three countries show high achievement in 
translation, computational applications of knowledge in geometry, and judgmental applications of 
knowledge in geometry. These students are low in applying and evaluating mathematical 
correctness. The profiles show that the largest differences between the U.S. and Korean eighth 
grade students are in logical reasoning and data management. The largest difference between the 
U.S. and the Czech Republic is in logical reasoning. 



Figure 3. Process Attribute Mastery Probabilities 
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Table 6. Process Attributes Mastery Probabilities 



Attribute 


Description 


U.S. 


Korea 


Czech 

Republic 


Korea- 

U.S. 


Czech 

Republic 

-U.S. 


PI 


PI: Translate/foniiulate equations 


0.95* 


0.97* 


0.98* 


0.02 


0.03 


P2 


P2: Computational applications of 
knowledge in geometry 


0.92* 


0.98* 


0.96* 


0.06 


0.04 


P3 


P3: Judgmental applications of 
loiowledge in arithmetic and geometry 


0.85* 


0.97* 


0.91* 


0.12 


0.06 


P4 


P4: Applying rules in algebra 


0.59 


0.76 


0.65 


0.17 


0.06 


P5 


P5: Logical reasoning 


0.65 


0.84 


0.82 


0.19 


0.17 


P6 


P6: Problem Search 


0.83 


0.93 


0.92 


0.10 


0.09 


P7 


P7: Generating Figures 


0.77 


0.93 


0.94 


0.16 


0.17 


P8 


P8: Applying & Evaluating mathematical 
correctness 


0.57 


0.59 


0.55 


0.02 


0.02 


P9 


P9: Management of Data & Procedures 


0.68 


0.95 


0.81 


0.27 


0.13 


PIO 


PIO: Quantitative & Logical Reading 


0.87 


0.88 


0.82 


0.01 


0.05 



*High achievement 



VI. Results 
U.S. vs. Korea 

Content A iiribuie ^Geometry ^ 

The common significant predictors are as follows: computer availability at home, how much time 
students spend outside school talking and playing with friends, students’ self-concept in 
mathematics and number of books in home. 

Although there is a clear positive association between computer availability at home and 
mathematics achievement inteniationally (Mullis, 1999), there is a negative association between 
computer availability at home and mathematics achievement in Korea. Tliere is a positive 
association between computer availability at home and achievement in the U.S. It may be that 
Korean students spend less time with computers for studying; rather they use computers for 
purposes such as gaming and internet surfing. The computer availability in U.S. homes is 
significantly higher than that of Korea. Higher self-concept in mathematics predicts higher 
achievement in geometry in Korea and the U.S., which means students with high mastery 
probability in Geometry would do well in other areas. The average self-concept in mathematics of 
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U.S. students is higher, in other words, more positive than the average of Korean students. The 
number of books in home is positively associated with geometry in both countries. With a 
significant difference represented, the number of books in a Korean home is higher than that of the 
U.S. The last common predictor in geometry is how much time students spend outside talking and 
playing with friends, which is negatively related to mastery probability in geometry. The time for 
playing of Korean students is significantly higher than the playing time of U.S. students. 



Table 7. Predictors of Achievement in Geometry 



Common predictors of achievement 
in Geometry 


Association 


Difference in 
Prediction 


Computer availability at home 


Positive (U.S) Negative (Korea) 


U.S. > Korea 


Students’ self-concept in math 


Positive 


U.S. > Korea 


Number of books at home 


Positive 


Korea > U.S. 


How much time students spend 
outside school playing with 
friends 


Negative 


Korea > U.S. 



Skill Attribute ^Open-Ended Item^ 



These regression analyses suggested that six common variables are significant predictors of 
achievement in open-ended items across the U.S. and Korea. The variables are as follows: number 
of books at home, how frequently the student listens to popular music, number of people living in 
the home, the degree at which the student is good at memorizing note, number of books at home, 
the degree to which the student has time for fun, and students’ self-concept in mathematics. 

The number of books at home is a positive predictor as described above section. Students’ self- 
concept in mathematics is positively related with the mastery probability of ‘Open-ended items’. 
U.S. students have higher efficiency than Koreans. How frequently the student listens to popular 
music is a predictor for die U.S. and Korea. A predictor shows socio-cultural differences between 
the U.S. and Korea. There is a negative association between them in the U.S,, but a positive 
association between them in Korea. The average frequency that U.S. students watch music shows is 
higher than the average of Korean with a significant difference. The number of people living in 
home is a negative predictor. In other words, it may be that students with more people in home have 
lower attribute mastery of Open-ended items. There are no significant difference of average number 
of people living at the home between U.S. and Korea. Tliere is a negative association between the 
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degree of playing with friends outside school and the mastery of open-ended items. The more 
playing time the lower mastery of open-ended items. There is a negative association between the 
degree to which the student has time for fun and the mastery probability of open-ended item. Tlie 
average time of Korean students is higher than the average time of U.S. students. There is a clear 
negative association between the degree to which the student is good at memorizing note and 
achievement in mastery of open-ended items. Thus, memorizing notes would be considered a 
hindrance of open-ended items that need a thinking process. U.S students more firmly believe they 
are good at memorizing notes than Korean do. 



Table 8. Predictors of Achievement in Open-ended Items 



Common predictors of achievement 
in open-ended items 


Association 


Difference in 
Prediction 


Degree at which the student agrees 
that he/she is good at memorizing 
note 


Negative 


U.S. > Korea 


Number of books at home 


Positive 


Korea >U.S. 


Number of people living at home 


Negative 


U.S. = Korea 


How frequently the student listens to 
popular music 


Positive (U.S.) Negative (Korea) 


U.S. > Korea 


How much time students spend 
outside school playing with friends 


Negative 


Korea > U.S. 



Process Attribute ^Loeical Reasoning* 



Common predictors in Logical reasoning are as follows: number of books in home, students’ self- 
concept in math, the degree to which students think math is more difficult for them than for others, 
preference for mathematics, and the highest education level of parents. 

The number of books in home is a positive predictor of logical reasoning in both countries. Tlie 
average of books at the home of Korea is significantly higher than the average of U.S. Self-concept 
in mathematics is a positive predictor in the U.S. and Korea. The students who think math is 
difficult have higher achievement on logical reasoning. U.S. students think that math is more 
difficult for them than for others when compared to Korean. Considering the achievement 
difference between both countries, lower mathematical self-concept would affect the 
accomplishment of logical reasoning. With a significant difference, the general efficiency of US 
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students is higher than Korean’s. The frequency of watching popular music video shows is again a 
common significant predictor of logical reasoning as well as open-ended items. The degree at 
which students believe they are good at memorizing predicts lower mastery of logical reasoning 
and open-ended items. As described above, U.S. student have higher average than Korean students. 
Preference for mathematics is positively related to the mastery probability of logical reasoning. 
High-perfonning Korea is the country with a less positive attitude than the U.S. It may be that the 
Korean students follow a demanding mathematics curriculum, one that leads to high achievement 
but little enthusiasm for mathematics (Mullis, 1999). There is a positive association with the 
highest education level of parents and student achievement in logical reasoning. Tlie results suggest 
that the higher education level of parents predicts better achievement of their children. The average 
level of education in US parents is higher than that of Korean parents. 



Table 9. Predictors of Achievement in Logical Reasoning 



Common predictors of achievement in 
logical reasoning 


Association 


Difference in 
Prediction 


Number of books at home 


Positive 


Korea > U.S. 


Students’ self-concept in math 


Positive 


U.S. > Korea 


Frequency of watching popular music video 
shows 


Negative (U.S.) Positive (Korea) 


U.S. > Korea 


Students’ positive attitudes towards math 


Positive 


U.S. > Korea 


Parents’ education 


Positive 


U.S. > Korea 



U.S. vs. Czech Republic 
Content Attribute ^Geometry* 

The results of these regression analyses suggested that four common variables appear to be 
significant predictors of attribute C4 mastery probabilities across the U.S. and the Czech Republic. 
Tliese variables are as follows: computer availability at home, students’ self-concept in math, 
student’s access to internet at home, and how much time the student spends outside talking and 
playing with friends. 

Tlie first two variables are positively related to attribute mastery and the remaining two are 
negatively related to attribute mastery both in the U.S. and in the Czech Republic data set. These 
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results are interesting in several ways. First thing to notice is that although computer availability is 
positively related to attribute mastery, internet access at home is negatively related to the same 
variable. It can be argued that while computer availability at home might be a reflection of one’s 
socio-economic status, having internet access at home gives a clearer idea about how the student 
uses the computer and therefore how he or she uses time at home. It is quite intuitive to assume that 
if one has internet access at home one will spend more time on surfing the web. This may lead to 
less time spent studying and with other school-related activities. It is not a secret that many young 
students spend hours on the web chatting with each other. On the other hand one of the other 
variables also supports the expectation that less time studying has negative effects on achievement. 
As stated above, the more time the student spends outside playing and talking to friends, the weaker 
his attribute mastery. These two results are quite parallel. It seems that time allocated to non- 
academic activities (whether it is spending time on the web at home or talking and playing with 
friends outside) has a negative effect on a student’s attribute mastery. The fourth variable, student’s 
self-concept in mathematics, is positively related to attribute mastery. A lower score on this 
variable indicates that the student agrees more with the idea that he or she is not talented in math. 
Tliis is also not surprising. A student’s self-perception about math obviously has an effect on his or 
her achievement. This is another context where this fact is observed: The more the student thinks he 
or she is not talented in math the weaker his or her attribute mastery. 



Table 10. Predictors of Achievement in Geometry 



Common predictors of achievement 
in Geometry 


Association 


Difference in Prediction 


Computer availability at home 


Positive 


U.S. = Czech Republic 


Internet Access at home 


Negative 


Czech Republic > U.S. 


Students’ self-concept in math 


Negative 


Czech Republic > U.S. 


How much time students spend 
outside school playing with 
friends 


Negative 


U.S. > Czech Republic 



Skill Attribute ^Open-Ended Item^ 



The results of these regression analyses suggested that six common variables appear to be a 
significant predictor of attribute SIO mastery probabilities across the U.S. and Czech Republic 
These variables are as follows; number of books at home, watch popular music video shows, 
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number of people living in the home, highest educational level of parents, the degree to which the 
student thinks he or she is not talented in math, and the degree at which the student agrees that 
he/she is good at memorizing notes. 

The first of these variables, number of books at home, is positively related to attribute mastery. This 
implies that the more books the student has at home the stronger his or her attribute mastery. Tliis 
result is not surprising in the sense that having more books at home might actually imply a couple 
of things which are all positively related to achievement: higher socio-economic status, a more 
academically-stimulating home environment, a parent with a higher educational level, or a parent 
who invests and appreciates education more. The second variable, watching popular music, is also 
positively related to attribute mastery. A higher score on this variable implies that the student 
watches less popular music shows on TV. Therefore it follows that the more the student watches 
popular music shows on TV, the weaker his or her attribute mastery. This result is also in line with 
several other factors discussed before. This variable-like some others- relates to how the student 
manages his or her time at home. It is argued above that the more the student has opportunity to 
access the internet, doing jobs, or talking and playing with friends outside the home, the weaker his 
or her attribute mastery. This last variable adds another aspect to the same topic: how the student 
manages his or her time -in and outside home- matters in determining his or her attribute mastery. 
The third variable, ‘number of people living at home,’ is negatively related to attribute mastery. 

That means that the more people there are at home the weaker the student’s attribute mastery. This 
might be because of a couple of different reasons. More people at home might mean less private 
space for the student, which may, in turn, mean less quite and a less free environment to study. Or 
the fact that there are more people at home might mean the student has more siblings than others, 
which in turn might result in less parental support and surveillance on student’s academic life. The 
fourth variable ‘highest educational level of parents’ is negatively related to attribute mastery. A 
lower score on this variable indicates a higher parental educational level. Therefore it follows that 
the higher the student’s parental educational level the stronger his or her attribute mastery. This 
result is also an expected one. Higher parental education might actually mean a couple of different 
things: higher socio-economic status, parents that appreciate and emphasize education more, and 
who can provide better support to the students for her academic needs. The fifth significant variable 
is the degree to which the student thinks he is just not talented in math. This variable is positively 
related to attribute mastery in both countries and actually appeared as a significant predictor for 
attribute C4 as well. Remember that a lower score on this variable indicates that the student agrees 
more with the idea that he or she is not talented in math. Therefore the more the student thinks he or 




17 



she is not talented in math the weaker his or her attribute mastery. The sixth and the last variable is 
the degree to which the student agrees that s/lie does well in memorizing class notes. A lower score 
on this variable indicates more acceptance of the idea that the student does well in memorizing 
notes. This variable is positively related to attribute mastery. Tliat means that the more the student 
thinks he or she does well memorizing notes the weaker his or her attribute mastery is. It is 
important at this point to remember that the attribute under investigation here is SIO, which is an 
open-ended item type. Mastery of this attribute requires the students to be more successful in open- 
ended item types. And these items require more skills in reasoning, proof, and simple memorizing. 
Therefore it is not surprising to see that students who rely on their memory skills more have more 
difficulty in mastering this attribute and answering open-ended items. 



Table 11. Predictors of Achievement in Open-ended Items 



Common predictors of achievement in open- 
ended items 


Association 


Difference in 
Prediction 


Degi'ee at which the student agrees that he/she 
is good at memorizing note 


Negative 


U.S. > Czech Republic 


Number of books at home 


Positive 


Czech Republic> U.S. 


Parents’ education 


Positive 


U.S. > Czech Republic 


How frequently the student listens to popular music 


Negative 


U.S. > Czech Republic 


Number of people living at home 


Negative 


U.S. > Czech Republic 


Students’ self-concept in math 


Negative 


Czech Republic > U.S. 



Process Attribute ^Losical Reasonin2^ 



Tliese variables are as follows: students’ self-concept in math, how much time the student spends 
doing jobs at home, and students’ attitudes towards mathematics. 

The first variable, the degree at which the student thinks math is more difficult for him or her than 
for others, is positively related to attribute mastery. On this variable a lower score indicates a 
greater acceptance of the afore-mentioned idea. Tlierefore the more the student agrees that math is 
more difficult for him or her than for others the weaker his or her attribute mastery. This result is 
another indication of the fact that students’ achievement is related to a self-perception about math. 
The second variable- how much time the student spends doing jobs at home- is negatively related to 
attribute mastery in both countries. A lower score indicates less time spent doing jobs at home. 
Therefore the results can be interpreted as follows: the more time the student spends in doing jobs 
at home the weaker his or her attribute P5 mastery probability. This result is very much in line with 
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what we found in analyzing attribute C5. In that analysis also we found that the more time students 
spend in non-academic activities (both at home and outside) the weaker their attribute masteries. 
Here again we found tliat spending more time on another non-academic activity at home also results 
in weaker attribute mastery. The third variable - how much the student reports he or she likes math- 
is also negatively related to attribute mastery in both countries. A lower score indicates a higher 
level of liking of math on this variable. Therefore the results suggest that the more the student 
reports he or she likes math the stronger his or her attribute P5 mastery. 

Table 12. Predictors of Achievement in Logical Reasoning 



Common predictors of achievement in logical 
reasoning 


Association 


Difference in 
Prediction 


How much time the student spends doing jobs at 
home 


Negative 


Czech Republic = U.S. 


Students’ positive attitudes towards math 


Positive 


U.S. > Czech Republic 


Degree at which the student thinks math is more 
difficult for him 


Positive 


Czech Republic > U.S. 



VII. Conclusion 

Diagnostic analyses were conducted of TIMSS-R data from the U.S., Korea, and the Czech 
Republic in terms of test item attributes (i.e., content, processing skills, and item format) and 
inferred knowledge states of students. 

U.S, Korean, and Czech Republic students show relatively high achievement in integer arithmetic 
skills, using figures and graphs, evaluation/verification, proportional reasoning, translation, 
judgmental applications of knowledge in geometry. Tliese students are low in statistics/data 
analysis, unit conversion, and applying and evaluating mathematical correctness. The largest 
differences between the U.S. and Korea are in geometry, data management, logical reasoning, 
patterns, and open-ended item. The largest differences between the U.S. and the Czech Republic are 
in geometry, logical reasoning, patterns, and open-ended items. Students’ self-concept in 
mathematics & socioeconomic composition are significant predictors of student achievement in 
geometry, open-ended items, and logical reasoning in the U.S., Korea, and the Czech Republic. 

We believe results of this study can be beneficial in improving the quality of classroom instruction 
in the U.S., Korea, and the Czech Republic and help in the areas of accountability and 
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policymaking. Our hope is that this study can add valuable insights into the influence of educational 
and curricular policy on mathematics achievement in Korea, the U.S., and the Czech Republic. 
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Appendix 1. A list of knowledge, skill, and process attributes derived to explain 
performance on the TIMSS-R (1999) math items. Population 2 (8^^ graders) 

(Tatsuoka, 2001). 

CONTENT ATTRIBUTES 

Cl Basic concepts, properties and operations in whole numbers and integers 

C2 Basic concepts, properties and operations in fractions and decimals 

C3 Basic concepts, properties and operations in elementary algebra 

C4 Basic concepts and properties of two-dimensional Geometry 

C5 Data, probability, and basic statistics 

C6 Using tools to measure (or estimating) length, time, angle, temperature 

PROCESS ATTRIBUTES 

PI Translate/formulate equations and expressions to solve a problem 
P2 Computational applications of knowledge in arithmetic and geometry 

P3 Judgmental applications of knowledge in arithmetic and geometry 
P4 Applying rules in algebra 

P5 Logical reasoning — includes case reasoning, deductive thinking skills, if-then, necessary 
and sufficient, generalization skills 

P6 Problem Search; Analytic Tliinking, Problem Restructuring and Inductive 
Thinking 

P7 Generating, visualizing and reading Figures and Graphs 
P8 Applying and Evaluating Mathematical Correctness 

P9 Management of Data and Procedures 

PiO Quantitative and Logical Reading 

SKILL (ITEM TYPE) ATTRIBUTES 

51 Unit conversion 

52 Apply number properties and relationships; number sense/number line 

53 Using figures, tables, charts and graphs 

54 Approximation/Estimation 

55 EvaluateA^ erify/Check Options 

56 Patterns and relationships (be able to apply inductive thinking skills) 

57 Using proportional reasoning 

58 Solving novel or unfamiliar problems 

59 Comparison of two/or more entities (deleted because of low frequencies in each booklet) 

SIO Open-ended item, in which an answer is not given 

SI 1 Using words to communicate questions (word problem) 
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